Why Unleavened Bread?

There are at least two reasons why God made the stipulation that bread intended for sacrifice at the altar had to be without yeast (or "unleavened"). We see one of the reasons in the depiction of the first Passover in Exodus 12:8: "That same night they are to eat the meat roasted over the fire, along with bitter herbs, and bread made without yeast." You may recall that this final meal for the Israelites in Egypt was a meal made speedily, specifically because the Israelites were to leave Egypt immediately once the Pharaoh relinquished his hold on them. Roasting the meat was an efficient way to deal with the fat, the bitter herbs were symbolic of the bitterness of their captivity, and the unleavened bread symbolized the haste in which the flight from Egypt was to take place (in that there was to be no time taken to allow the bread to rise).

A few verses farther on (Exod. 12:15) we see another significant aspect of the symbolism of yeast as it relates to your question. As part of the observance of Passover, purification of the household was symbolized by the removal of all leaven from the house. (In fact, this is still observed in Jewish Passover observances today, often the duty of the children.) Leaven, thus, is frequently understood in Scripture to be symbolic of evil. Christ clearly alludes to this symbolism in Matthew 16:6 ("the yeast of the Pharisees and Sadducees"). The Apostle Paul in 1 Corinthians 5:6 extends and further illustrates the symbolism in saying, "Don't you know that a little leaven works through the whole batch of dough?" in his instructions to expel from the Corinthian congregation a man who had a sexual relationship with his stepmother. The message is clear: Continued contact of this person with the congregation would spread the influence of evil throughout the church. See also the Q&A Yeast for further information.

As a side note, the Hebrew and Greek in these passages refer to "leaven" rather than to "yeast" (see the last paragraph of Do the Elements of Communion Matter? for the distinction between "leaven" and "yeast").